Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle issues introduced by OTLP gRPC protocol #170

Merged
merged 3 commits into from
Apr 22, 2024
Merged

Handle issues introduced by OTLP gRPC protocol #170

merged 3 commits into from
Apr 22, 2024

Conversation

thpierce
Copy link
Contributor

@thpierce thpierce commented Apr 22, 2024

In this commit, we are handling issues that arise from gRPC. Essentially, if we build gRPC artifacts into our Docker image, it causes the Docker image to only be compatible with applications built using the same Python version. To solve this, we are doing two things: 1) we are removing gRPC artifacts from the docker image and 2) we are changing the default OTLP protocol to be HTTP. If customers attempt to set the protocol as gRPC for ApplicationSignals, we will set the default endpoint correctly.

Also we are changing Docker image to build with Python 3.11, which is what we were originally doing when we encountered this issue (reference: 5b3ed74). This is what the upstream does (see autoinstrumentation/python/Dockerfile), and having parity here is beneficial to us.

Testing:

  • Create app.py:
from time import sleep
import boto3

try:
    boto3.client('s3').list_buckets()
except Exception:
    sleep(100)

  • Run ./scripts/build_and_install_distro.sh
  • Run:
export OTEL_PYTHON_DISTRO="aws_distro"         
export OTEL_PYTHON_CONFIGURATOR="aws_configurator"
export OTEL_METRICS_EXPORTER="none"
unset OTEL_EXPORTER_OTLP_PROTOCOL
unset OTEL_AWS_APPLICATION_SIGNALS_ENABLED
  • Run opentelemetry-instrument python ./app.py
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc647cd5e50>: Failed to establish a new connection: [Errno 111] Connection refused'))
  • Run export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; opentelemetry-instrument python ./app.py
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f20ac96e490>: Failed to establish a new connection: [Errno 111] Connection refused'))
  • Run export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument python ./app.py
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s.
  • Run
unset OTEL_EXPORTER_OTLP_PROTOCOL
export OTEL_METRIC_EXPORT_INTERVAL=1000
export OTEL_AWS_APPLICATION_SIGNALS_ENABLED=True
  • Run opentelemetry-instrument python ./app.py
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4316): Max retries exceeded with url: /v1/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8f6c5aa5b0>: Failed to establish a new connection: [Errno 111] Connection refused'))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8f6dd9c9d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
  • Run export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf; opentelemetry-instrument python ./app.py
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5868efdc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5868efdc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
  • Run export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument python ./app.py
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4315, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@thpierce thpierce requested a review from a team as a code owner April 22, 2024 16:47
In this commit, we are handling issues that arise from gRPC.
Essentially, if we build gRPC artifacts into our Docker image, it causes
the Docker image to only be compatible with applications built using the
same Python version. To solve this, we are doing two things: 1) we are
removing gRPC artifacts from the docker image and 2) we are changing the
default OTLP protocol to be HTTP.
@thpierce thpierce merged commit 80be26b into main Apr 22, 2024
11 checks passed
@thpierce thpierce deleted the grpc branch April 22, 2024 18:35
thpierce added a commit that referenced this pull request Apr 22, 2024
In this commit, we are handling issues that arise from gRPC.
Essentially, if we build gRPC artifacts into our Docker image, it causes
the Docker image to only be compatible with applications built using the
same Python version. To solve this, we are doing two things: 1) we are
removing gRPC artifacts from the docker image and 2) we are changing the
default OTLP protocol to be HTTP. If customers attempt to set the
protocol as gRPC for ApplicationSignals, we will set the default
endpoint correctly.

Also we are changing Docker image to build with Python 3.11, which is
what we were originally doing when we encountered this issue (reference:
5b3ed74).
This is what the upstream does (see
[autoinstrumentation/python/Dockerfile](https://github.com/open-telemetry/opentelemetry-operator/blob/b5bb0ae34720d4be2d229dafecb87b61b37699b0/autoinstrumentation/python/Dockerfile)),
and having parity here is beneficial to us.

Testing:
* Create `app.py`:
```
from time import sleep
import boto3

try:
    boto3.client('s3').list_buckets()
except Exception:
    sleep(100)

```
* Run `./scripts/build_and_install_distro.sh`
* Run:
```
export OTEL_PYTHON_DISTRO="aws_distro"         
export OTEL_PYTHON_CONFIGURATOR="aws_configurator"
export OTEL_METRICS_EXPORTER="none"
unset OTEL_EXPORTER_OTLP_PROTOCOL
unset OTEL_AWS_APPLICATION_SIGNALS_ENABLED
```
* Run `opentelemetry-instrument python ./app.py`
```
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fc647cd5e50>: Failed to establish a new connection: [Errno 111] Connection refused'))
```
* Run `export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf;
opentelemetry-instrument python ./app.py`
```
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f20ac96e490>: Failed to establish a new connection: [Errno 111] Connection refused'))
```
* Run `export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument
python ./app.py`
```
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s.
```
* Run
```
unset OTEL_EXPORTER_OTLP_PROTOCOL
export OTEL_METRIC_EXPORT_INTERVAL=1000
export OTEL_AWS_APPLICATION_SIGNALS_ENABLED=True
```
* Run `opentelemetry-instrument python ./app.py`
```
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4316): Max retries exceeded with url: /v1/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8f6c5aa5b0>: Failed to establish a new connection: [Errno 111] Connection refused'))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8f6dd9c9d0>: Failed to establish a new connection: [Errno 111] Connection refused'))
```
* Run `export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf;
opentelemetry-instrument python ./app.py`
```
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5868efdc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4318): Max retries exceeded with url: /v1/traces (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff5868efdc0>: Failed to establish a new connection: [Errno 111] Connection refused'))
```
* Run `export OTEL_EXPORTER_OTLP_PROTOCOL=grpc; opentelemetry-instrument
python ./app.py`
```
Transient error StatusCode.UNAVAILABLE encountered while exporting metrics to localhost:4315, retrying in 1s.
Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s.
```

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
thpierce added a commit that referenced this pull request Apr 22, 2024
In this PR, we are bringing in patches to resolve an issue introduced by
depending on OTLP gRPC protocol. Please see
#170
for more details.

By submitting this pull request, I confirm that you can use, modify,
copy, and redistribute this contribution, under the terms of your
choice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants